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Executive summary 


The project 


Thinking, Doing, Talking Science (TDTS) is a programme that aims to make science lessons in primary 
schools more practical, creative and challenging. Teachers are trained in a repertoire of strategies that 
aim to encourage pupils to use higher order thinking skills. For example, pupils are posed ‘Big 
Questions’, such as ‘How do you know that the earth is a sphere?’ that are used to stimulate discussion 
about scientific topics and the principles of scientific enquiry. 


Two teachers from each participating school received five days of professional development training 
delivered by a team from Science Oxford and Oxford Brookes University. The training did not aim to 
provide participating teachers with a set of ‘off-the-shelf lesson plans to be delivered in schools; rather, 
it sought to support teachers to be more creative and thoughtful in planning their science lessons. In 
addition, teachers had dedicated time to work with colleagues to plan and review lessons taught as part 
of the project. Teachers were also encouraged to link pupils’ learning in science, with their learning in 
numeracy and literacy. 


This project sought to assess the impact of the programme on the academic outcomes and attitudes 
towards science of Year 5 pupils. 655 pupils from 21 schools across England completed the project. 
Participating schools followed the programme for the entirety of the 2013/14 academic year. A further 
20 schools formed a randomised comparison group and did not receive training in the approach until 
the following year. 


Key conclusions 


1. Thinking, Doing, Talking Science appeared to have a positive impact on the attainment of 
pupils in science. Overall, Year 5 pupils in schools using the approach made approximately 
three additional months’ progress. 

2. There are some indications that the approach had a particularly positive effect on pupils 
eligible for free school meals, but further research is needed to explore this. 


3. The programme had a particularly positive effect on girls and on pupils with low prior 
attainment. 


4. The approach had a positive impact on pupils’ attitudes to science, science lessons, and 
practical work in particular. 

5. National test data will be used to assess the English and mathematics outcomes of 
participating pupils and to measure the long-term impact of the approach. In addition, further 
research could be conducted to investigate whether this result can be replicated in a larger 
number of schools. 


Security rating awarded as part 
Security rating of the EEF peer review process 


Findings from this evaluation have moderate security. The study was set up as a randomised controlled 
trial, which aimed to compare the progress of pupils who received the programme to similar pupils who 
did not. The trial was classified as an efficacy trial, meaning that it sought to test whether the intervention 
can work under ideal or developer-led conditions in ten or more schools. 


There was very low drop-out from the project (only 1 school out of the initial 42 who signed up), and the 
participating pupils appeared to be very similar to those in the comparison group. It is unlikely that the 
observed result occurred due to chance. 


In the absence of a nationally recognised science assessment, a test was developed using age- 
appropriate, curriculum-relevant questions that had previously been used in a similar study. The tests 
were administered by participating teachers, who did not have access to the test prior to the day that it 
was taken. The security rating is discussed further in Appendix 9 of the main report. 
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Results 


e On average, participating pupils achieved higher scores on a combined measure of science 
knowledge and understanding, compared to pupils in the comparison group. The effect was 
equivalent to roughly three additional months’ progress. 

e There appeared to be greater effects for girls (+4 months on average) than for boys (+2 months). In 
addition, it is not possible to rule out chance as an explanation of the positive effect for boys. 

e There appeared to be a slightly greater positive effect for pupils with low prior attainment (+4 months) 
compared to those with high prior attainment (+3 months). However, all pupils benefitted relative to 
similar children in comparison schools. 

e Pupils eligible for free school meals made five additional months’ progress on average, compared to 
similar pupils in comparison schools. However, due to the low numbers of pupils eligible for free 
school meals participating in the trial, this finding has lower security. 

e Pupils in classes following the approach reported having more positive attitudes towards science on 
the vast majority of measures, and were more likely to believe that it was important to learn science. 

e In line with the TDTS model, pupils in classes following the approach reported having more 
discussions and doing less writing in lessons, compared to those in comparison classes. 

e Participating teachers reported using practical work, discussions and time for thinking much more 
often than teachers in comparison schools. There was little evidence that participating teachers were 
more confident than teachers in comparison classes. 

e All participating teachers felt that they had changed the way they taught science, and were more 
positive about their pupils’ science ability and engagement than teachers in comparison schools. All 
teachers were positive about their experience of the training sessions and resources. 


Cost 


The cost of the approach as delivered in the trial is estimated at £26 per pupil. This estimate is based 
on two classes of 25 pupils and a group of 21 schools receiving training together. Schools were charged 
£1,000 each to take part, which included five full INSET days, an in-school ‘launch day’ to begin the 
project, resources and materials. The estimate also includes the cost of three pilot days, which were 
used to refine the approach prior to the start of the intervention but were not passed onto schools 
(estimated at £6,720 in total). 


Group Effect size Estimated Security rating 
months’ progress 


All pupils +3 months &4a8 
Girls +4 months 


Boys +2 months 


FSM pupils +5 months 


Education Endowment Foundation 5 


Thinking, Doing, Talking Science 


Introduction 


Intervention 


Thinking, Doing, Talking Science (TDTS) comprises a series of professional development sessions, 
support, and resources for primary school teachers. It aims to improve pupils’ thinking skills and science 
attainment by making science lessons more conceptually challenging, more practical, and more 
interactive. Rather than teaching science simply as a body of facts to be learned, the approach 
emphasises the principles of scientific inquiry: how to ask good questions and design simple 
experiments to find out the answers. The aim is not to provide a set of ‘off-the-shelf lesson plans to be 
delivered in schools; rather it is to make teachers more creative and thoughtful in planning their science 
lessons and to equip them with a repertoire of strategies and ideas that they can use and adapt to suit 
their context. 


The project included a pilot stage in the previous academic year, during which Year 5 teachers in the 
intervention schools received three days of professional development. This helped refine the training 
sessions and introduced the approach to the school (though importantly, not to the study cohort of 
pupils). In the main study, two teachers from each school participated in five days of professional 
development (both were Year 5 teachers or, in one-form entry schools, Year 5 and another member of 
staff such as the science subject leader). Due to staff changes, in many cases these were not the same 
teachers that had taken part in the pilot. In the professional development sessions, the teachers were 
shown how to plan and deliver their own conceptually challenging lessons, tailored to the needs and 
abilities of their pupils. They also spent two days in-school planning and sharing knowledge to ensure 
the strategies were embedded. 


Background evidence 


The evidence that this approach can work comes from a quasi-experiment in primary schools in rural 
Oxfordshire (Mant, Wilson and Coates, 2007). In that study, 16 treatment schools were matched with 
16 control schools based on size, results in previous science SATS, and proportion of children with 
special educational needs. With intensive training, the treatment schools used the ‘conceptually 
challenging’ approach for one year. Researchers then compared the improvement in Key Stage 2 
science results across the two groups. They found that the proportion of pupils achieving level 5 in 
science increased by 10 percentage points more in treatment schools than in control schools. 


The approach is an attempt to remedy a major problem with primary level science teaching highlighted 
by the Parliamentary Office for Science and Technology (2003): primary school teachers tend to 
concentrate too much on factual content and not enough on core scientific ideas. The EEF decided the 
project was suitable for an efficacy trial, using random assignment to get a better estimate of the impact 
and to test whether the approach works in disadvantaged schools. It was the first science project funded 
by the EEF. 


Evaluation objectives 


The evaluation was designed to establish whether TDTS had any impact on: 


e pupils’ science attainment; 
e pupils’ attitudes towards science; and 
e pupils’ performance in English and Maths at KS2. 


Project team 


The TDTS programme was developed and delivered by Bridget Holligan from Science Oxford and Helen 
Wilson from the School of Education, Oxford Brookes University. 


Education Endowment Foundation 6 


Thinking, Doing, Talking Science 


Ethical review 


The evaluation team obtained ethical approval from the Department of Education, University of York 
Ethical Review Panel on 27 July 2012. Headteachers signed an agreement outlining the main 
commitments of the three parties in the study: the school, the project developers and the evaluators. 
The evaluation team provided information and opt-out consent forms for parents/guardians. 


Data was managed in accordance with the Data Protection Act (1998). The trial database is securely 
held and maintained on the University of York’s research data protection server, with non-identifiable 
data. Confidentiality is maintained and no one outside the trial team has access to the database. Data 
was checked for missing elements and/or double entries. All outputs were anonymised so that no 
schools or students could be identified in any report or dissemination of results. 


Trial registration 


This trial was registered at http://www. isrctn.com/ISRCTN54969918 
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Methodology 


Trial Design 


A two-armed clustered randomised-controlled trial was carried out. Forty-two primary schools were 
recruited to the project in Autumn 2012. The evaluators matched schools into pairs based on pre-test 
results, pupils eligible for free school meals, pupils with English as an additional language, and the size 
of the school. One of each pair was randomly allocated to receive TDTS and the other to the delayed 
treatment control. 


School-level randomisation was chosen in preference to within-school randomisation (e.g. within-year 
randomisation in two-form entry schools) because teacher collaboration is inherent to the intervention. 
Therefore, there would have been a high risk of teachers in the comparison groups learning about the 
intervention approaches, for example through staffroom conversation or witnessing classroom practice. 
A wait-list control group was used to prevent teachers becoming demoralised post-allocation and to 
keep them engaged in the trial. 


Between January and June 2013, Year 5 teachers in the intervention schools were introduced to the 
TDTS programme to implement it with their current pupils (who were not part of the main study). From 
September 2013 to July 2014, TDTS was delivered to the same 21 intervention schools. The 21 control 
schools were offered the intervention from September 2014 to July 2015. 


Both groups were pre- and post-tested (in December 2012/January 2013 and June/July 2014 
respectively). The self-completion tests were administered by the teachers, with the pre-test being 
completed before randomisation (when the pupils were still in Year 4). The post-test marking was 
blinded, meaning that the assessors were unaware of which treatment group the pupils were in. The 
analyses used an intent-to-treat-design, meaning that every effort was made to retain schools that 
dropped out for any reason in the main analyses. 


Eligibility 


All primary schools in Oxfordshire were eligible for recruitment to this trial provided they were large 
enough to release a pair of teachers for the training and had not taken part in the previous related study 
(Mant et al., 2007). They were approached by the developers and invited to a recruitment conference 
where they learnt about TDTS and where the concept and practicalities of participating in an RCT were 
explained to them. At least one of the teachers agreeing to participate had to be teaching science to 
Year 5 and the cohort of pupils in Year 5 during the 2013-2014 school year was involved in the 
evaluation. 


Schools committed themselves to meet the requirements of the programme and evaluation when they 
signed up to the trial, and opt-out consent for pupils to take part in pre- and post-testing was sought 
from parents/guardians via the schools. 


Intervention 


TDTS is a primary school CPD programme designed to help teachers plan challenging science lessons 
that will enhance children’s engagement and cognitive skills. After three preliminary sessions to refine 
the content of the training days in the 2012-2013 academic year, the main intervention consisted of a 
series of five full-day sessions over the 2013-2014 academic year. The main five sessions took place 
at Oxford Brookes University premises in September and November 2013, and January, March and 
June 2014. Details of the content of these sessions can be found in Appendix 2. The training days were 
designed to enable the teachers to develop: 


e their questioning skills, to extend pupils’ thinking about scientific ideas; 
e dedicated discussion slots in science lessons; 
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e their understanding of appropriate and challenging science practical work, including 
investigations and problem solving; and 
e pupils’ focused and creative recording in science. 


The CPD was spread over the academic year to allow teachers to try out and evaluate the suggested 
strategies with their pupils between sessions. It was delivered by Bridget Holligan from Science Oxford 
and Helen Wilson from Oxford Brookes University. Over half (22 out of 38) of the teachers who attended 
the main phase of training sessions reasonably regularly also attended the three preliminary sessions. 


Prior to the main intervention, a launch day was delivered in each intervention school to demonstrate 
approaches to science teaching and to enthuse the school more generally about the project. The launch 
day was delivered to children who were not going to be involved in the study (meaning that they would 
not be in Year 5 in 2013-2014). In three schools, the launch days were delayed until September/October 
2013. The event involved pupils (not in the evaluation cohort) participating in a hands-on investigation, 
for example trying to identify the best ketchup for a café to use. 


The key features of the CPD were focused on three main elements: thinking, talking and doing science. 
This included the use of a dedicated discussion slot, the ‘Bright Ideas Time’, which utilised specific 
prompts to stimulate pupil talk, in particular: 


e The Odd One Out 
e The PMI 
e The Big Question 


In the first of these, the pupils are shown three different objects and asked to say which the odd one out 
is and why. This gives them the opportunity to draw upon their scientific understanding and vocabulary 
to justify their answer. There is no one ‘right’ answer. 


The ‘PMI thinking tool’ (De Bono, 1995) presents a scenario and asks pupils to identify and discuss a 
positive (P), a minus (M) and interesting features (I). So, for example, if the scenario was ‘all door 
handles are made of chocolate’, the pupils would need to use their understanding of the properties of 
materials to give pros and cons, and then move on to creative, imaginative ideas that arise from the 
scenario. 


The ‘Big Question’ poses a question linked to an area of science that will extend the children’s thinking. 
For example, ‘How do you know the Earth is a sphere?’ 


Each training day was themed around a particular area of the science curriculum, such as evolution, 
forces, or electricity. Teachers were given specific examples of each of these prompts in the different 
areas of the science curriculum and were encouraged to always include at least one in their science 
lessons. 


Another strategy was the use of ‘Practical Prompts for Thinking’—short teacher demonstrations that are 
designed to intrigue pupils and act as discussion starters. 


There was also an emphasis on pupils undertaking a variety of different types of practical science, 
including problem solving and fair testing. The pupils focused their recording on the learning objectives, 
thus trimming the time spent writing and releasing time for thinking, talking and doing. The aim was to 
make it sharp and focused so that it would be useful as an assessment tool for teachers. 


The control schools continued with business-as-usual teaching. They were all offered the training on 
completion of the evaluation, starting in September 2014, with launch days as for the intervention 
schools. 
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Outcomes 


The primary outcome was knowledge, thinking and reasoning in science. Since KS2 science SATs were 
discontinued in 2008, there has been no widely-recognised standardised science assessment for 
English primary pupils of any age group. The evaluators developed tests that spanned science 
curriculum content appropriate for the year group tested. These tests were compiled from standardised 
assessment questions' and had actually been originally devised for a previous, unrelated RCT?. Each 
test was carefully compiled to represent questions from a range of topics (broadly biology, chemistry 
and physics) and a balance of different types of assessment question (process/inquiry-based; shorter 
concept-based; and more open-ended conceptually-based). Copies of the tests can be found in 
Appendices 5 and 6. 


The secondary outcome reported here was pupils’ attitudes towards science, measured by a 
questionnaire adapted from Kind, Jones and Barmby (2007). The survey instrument can be found in 
Appendix 7. An additional secondary outcome will be attainment in English and Maths. This will be 
measured by obtaining the study pupils’ KS2 results in autumn 2015. (An analysis of those figures will 
be published as a later addendum to this report). [Addendum added in July 2016, see appendix 11] 


The tests and attitude surveys were posted to schools and completed under exam conditions, 
supervised by teaching staff in school. The pre-testing took place before randomisation, but the staff 
were aware of allocation during post-testing. Data marking and entry was carried out by IEE staff who 
were unaware of which treatment group the pupils were in. 


Sample size 


The statistical power of the planned analyses was estimated using Optimal Design software. Based on 
extensive experience with similar analyses, assumptions were made as follows: 

Pupils per school per year group: 45 

Proportion of variance in the outcome explained by covariates (R-squared): +0.49 

Intra-class correlation: 0.10 

Criterion for statistical significance (a): p<.05 

MDES: 0.25 

Power: 0.80 


Based on these assumptions, a sample size of 40 schools would be needed (20 per treatment group) 
to detect an effect size of 0.25. The recruitment target was 46 schools to allow for some dropout. The 
total number of schools that signed up to the study was slightly lower than this, at 42. The power 
calculations at different stages of the project (as planned in the protocol; at randomisation; and at 
analysis) are shown below. 


1 We are grateful to Terry Russell and Linda McGuigan (Centre for Research in Primary Science and Technology, University of 
Liverpool)for permission to take these questions from Science Assessment Series 1/2 
2 Wellcome Trust, publication pending 
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Correlation a 
N between pre- Bundi 
Peenoole puis] test (+other Power Alpha Getectable 
(n=intervention; effect size 


n=control) sae & (MDES) 


Protocol 40 (20;20) 
45 pupils per 


school 


Randomisatio 42(21;21) 

n 45 pupils per 
school 

Analysis (i.e. 41 (21;20) 

available pre- 23 pupils per 

and post-test) school 


Randomisation 


The schools were allocated to matched pairs based on the following criteria: pre-test score; students 
eligible for free school meals (FSM); school size, and students with English as an additional language 
(EAL). A random number generator was used in Excel to randomise one of each pair to the control and 
one to the intervention. The randomisation was carried out by the IEE’s Data Manager. 


Analysis 


The impact evaluation of the pupil tests used hierarchical linear modelling (HLM), a multilevel analysis 
in which pupils are nested within schools. Because randomisation was carried out using matched pairs, 
the pairings were accounted for by including them as an extra level in the analysis. This results in a 3 
level model, with Pupil nested within School nested within Pair. The HLM analysis used degrees of 
freedom associated with the number of schools, not the number of students. Pupils in schools randomly 
assigned to TDTS were compared to those in the randomly assigned control group, controlling for the 
pre-test scores. After the main analyses across all pupils, subgroup analyses were carried out for boys 
and girls; for high and low achievers based on the pre-test scores (using greater or equal to the median 
and below the median). Although we conducted an analysis for FSM-eligible pupils (defined as ever 
eligible for FSM), it should be noted that there were four schools with no pupils eligible for free school 
meals, and nine further schools with fewer than five FSM pupils. There were a number of reasons for 
this low number of FSM pupils. Firstly, there was a focus on poorly-performing schools and not just 
those with high rates of FSM at recruitment. Secondly, more of the schools recruited through the 
developers’ existing contacts had low rates of FSM. Thirdly, reflecting the rural nature of parts of the 
county, some schools were very small and did not have many Year 5 pupils in total. Since there was 
such a high level of attendance at the training sessions, which acted as a proxy for implementation 
fidelity, it was not appropriate to perform the intended analysis by fidelity. 


The effect size was calculated using Hedges’ g. Since this was a cluster randomised trial, there are 
options for the calculation of the effect size (Hedges 2007). We chose to calculate the effect size as the 
adjusted difference in the means between the arms (accounting for pairing) divided by the unadjusted 
total standard deviation in the outcome at endline. The pupil attitude ratings were analysed using 
descriptive statistics (paired comparison of school-level means for each statement). 


Despite requests from the evaluation team, it proved impossible to persuade one control school to 
complete the post-tests. However, there was no important difference to the estimated values when 
including or excluding the intervention school with the missing paired control school. Therefore it was 
included in the analysis, to use all the available data. 
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Process evaluation methodology 


A process evaluation was conducted to explore intervention and control teachers’ approaches to science 
teaching and intervention teachers’ feedback about TDTS. This was achieved through a short self- 
completion survey dispatched at the same time as the pupil post-tests and attitude surveys. In the 
intervention schools, staff who had attended the TDTS CPD sessions were asked to complete these, 
and in the control schools copies were addressed to Year 5 teachers. Attendance records from the 
training sessions in the evaluation year were also studied. There were no school visits in this study. 


The teacher survey was conducted with teachers in both treatment and control schools. It was a postal 
survey and recipients were assured that data would remain anonymous. Teachers were asked to 
indicate only the name of their school, although they could have returned it completely anonymously 
had they wished. Questions covered teaching approaches used, confidence in different aspects of 
science teaching and reflection on their pupils’ engagement with and progress in science across the 
year. These questions were identical for intervention and control teachers so that comparisons could be 
made. Intervention teachers were then asked additionally about TDTS: the training, the perceived effect 
on their classroom practice and their pupils; and which of the strategies they found most and least useful. 
A copy of the questionnaire can be found in Appendix 6. 
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Impact evaluation 


Timeline 


The main intervention period was the 2013-2014 academic year, when the study pupils were in Year 5, 
and taught science by participating teachers. Pre-tests were administered when the study cohort was in 
Year 4. The wait list control schools received the intervention from September 2014, with specific 
instruction not to use TDTS with the study cohort (now in Year 6). In May 2015, participating pupils 
completed their Key Stage 2 SATs tests. These will subsequently be analysed through the National 
Pupil Database. 


Date Activity 
October-December 2012 Recruitment of schools. Parental opt-out consent sought 


December 2012-mid January 2013 __Pre-tests 


23 January 2013 Randomisation 

February 2013-July 2013 Intervention (pilot stage) 

September 2013-July 2014 Intervention (main stage) 

June-July 2014 Post-tests 

June-July 2014 Pupil attitude questionnaires 

June-July 2014 Teacher surveys 

September 2014-July 2015 Waitlist control schools receive TDTS (not study pupil 
cohort) 

May 2015 Pupils sit KS2 SATs 


Participants 


Recruitment was managed by the development/delivery team. There were 235 primary schools in 
Oxfordshire at the start of the project, and it was estimated that about 100 would be ‘ineligible’ either 
because they were very small and would probably find it difficult to participate in the study as envisaged, 
or because they had been part of the previous, related research with Oxford Brookes University. 


The developers contacted 88 schools to ask if they would participate in the project. Of those, 24 said 
‘no’; 22 failed to reply to repeated attempts at contact (combinations of phone, e-mail and fax); and 42 
agreed to take part. Although the aim was to recruit 46 schools to allow for attrition, the high retention 
rate meant that there were more than the required 40 schools remaining at post-test. The developers 
struggled with the funding model (see Costs section). Asking for money from the schools to participate 
in the trial acted as a barrier to recruitment because it made it much harder to explain the project benefits 
and funding procedures to the headteachers. Some schools were also not in a position to produce the 
necessary funds and other arrangements had to be made so that they were not deprived of the 
opportunity to take part. 


The first 30-40 schools contacted were selected on the basis of appearing to be a good match to EEF 
‘target schools’, meaning that they had a high number of FSM students and below average performance 
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data. The second 40-50 were schools known to Oxford Brookes University through existing 
partnerships. Recruitment took place between October and December 2012. 


The evaluation team sent opt-out consent letters to all 42 schools in the trial for dispatch to 
parents/guardians of pupils in the trial. Parents were given a form to return to the school if they did not 
want their children to participate in the pre- and post-tests. No forms were returned to the evaluation 
team. 


The pupils completed the pre-test when they were in Year 4 and the post-test at the end of Year 5 (18 
months later). Post-tests were dispatched along with a list of pupils that had completed the pre-tests, to 
ensure that only the correct pupils were asked to complete the test. One control school (38 pupils) 
withdrew because, despite contact from the evaluators and the developers, the new head teacher could 
not find time for the pupils to complete the post-test. A further 211 pupils (108 intervention and 103 
control) had left the schools or were absent at the time of the post-test. This was an attrition rate of 2% 
of schools and 16% of pupils (see Figure 1). 


Randomisation was carried out by the IEE’s Data Manager using a random number generator. Schools 
were allocated to matched pairs based on pre-test score; pupils eligible for free school meals (FSM); 
school size; and pupils with English as an Additional Language (EAL). One of each pair was randomised 
to the control and one to the intervention using a random number generator. Table 1 shows that the 
schools were well matched in terms of size, type and level of FSM and EAL, at randomisation and in the 
final analysis. The proportion of pupils with FSM (15-16%) was slightly lower than the national average 
of 19%, as was the proportion with EAL (16% compared with 18% nationally, DfE, 2013). The group of 
intervention schools was more likely to be based in an urban or rural environment, and the control 
schools in the town and fringe. Although it was not controlled for in the analysis, it should be noted that 
almost half the intervention schools were rated as requiring improvement or inadequate in the Ofsted 
inspection prior to recruitment, compared with none of the control group. 
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Table 1: School characteristics 


As randomised (N=42) As analysed (N=41) 


Number of schools a 
Number on roll — 56 —_ 
(average) 


=} — ee 
School Type 


a cE 
LEA Maintained 81% 
Academies 19% 


Be ar ac 
SchoolSetting | SC ee as 
Urban at% 
Town and Fringe 0% 
Village 19% 


Mixed school 100% 
Eligible FSM 16.0% 
EAL 16.3% 


at (eae 

Ofsted Rating es 

Outstanding 0 i ie 

Good m1 | 7 | 
improvement 

Inadequate a 2 oe 
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Figure 1: Sample allocation and attrition 
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Approached (school n =88) 


Declined to participate 
(school n=24) 


Excluded (school n=0) 
Not meeting inclusion 
criteria (school n=0) 


Other reasons (school 
n=22) 
Randomised 
(school n=42; pupil n=1513) 


Allocated to control 
(school n=21; pupil 
n=750) 

Did not receive 
allocated intervention 
(school n=0; pupil 
n=0) 


Post-test data Lost to follow-up 
collected (pupil (pupil n=141) 
n=609) Moved _ schools/ 
unavailable for 
testing pupil 
n=103 

School withdrew 
(school n=1; pupil 
n=32A) 


Analysed (school Not analysed 
n=20; pupil n=609) (school n=1; 
pupil n=141) 
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Pupil characteristics 


Table 2 summarises the pupil characteristics of the sample, comparing the pre- and post-test profiles. 
At pre-test, the average score for pupils subsequently allocated to the intervention and control groups 
was almost identical. The control group had a higher proportion of boys than girls, whereas the 
intervention group was fairly evenly split. At 23% (intervention) and 22% (control), the proportion of 
pupils ever eligible for FSM was similar in the two groups. “Ever eligible” is those who have ever claimed 
FSM and is therefore higher than the figure shown in Table 1, which shows those currently eligible and 
claiming. The profile for pupils completing post-tests was very similar to the pre-test, indicating that 
attrition had acted at a similar rate across these demographics. 


Table 2: Pupil characteristics 


All pupils in 42 reporting schools All pupils with post-test scores 
(as randomised) (as analysed) 


Control Difference | Intervention Control Difference 
Pre-testscore | 24.04 | 2403 | 0.01 24.17 24.26 
Gender 


Male 48.1% 49.9% 50.1% 53.9% -3.8% 
Female 47.1% 41.5% 49.6% 45.3% 4.3% 


FSM (Ever FSM) 


No 71.8% 69.9% 1.9% 17.3% 76.5% 0.8% 


Yes 23.3% 21.5% 22.4% 22.7% -0.3% 
Number of 763 750 655 
pupils (all) 


Outcomes and analysis 


e)) 
[o) 
© 


For analysis there were 1264 completed post-tests, representing 84% of the original children. 


The outcome variable was tested to establish if there was any difference between intervention and 
control in a three step modelling process: 


Statistical analysis showed that both ‘pair’ and ‘school’ were, as expected, necessary components of 
the multilevel model. As the schools were chosen to be representative of all schools available to the 
study, both ‘pair’ and ‘schools’ were included in the model as random factors. 


Next the effect of the baseline score for each child was added to the model. This controlled the outcome 
scores for the pre-test, showing how important the pre-test was in predicting outcome scores, given the 
school the child is in. Pre- and post-test marks were strongly correlated: r= .71, p< .001. 


We then checked whether the effect of the pre-test score on the post-test score was the same for all 
schools, or varied between schools. Including this effect in the model did not improve the fit. So the 
effect of the pre-test score did not vary between the schools. 
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We fitted and tested the final multilevel model, which accounted for schools and pairs as random factors, 
and adjusted for baseline pre-test marks. This model tests the effect of the Intervention against the 
Control after the pre-test measure had been accounted for and taking into account the multilevel 
structure of the data. We assessed the diagnostics for this model and found them to be adequate, 
therefore we are confident that this model is robust. 


The post-test marks for the Intervention, after taking account of the pre-test mark and the variability 
among the schools, were 1.5 units higher than the control [95% Cl (0.74, 2.25)]. This result is unlikely 
to happen by chance [F(1, 36.50) = 6.61, p = .014]. 


The analyses performed for the full data were repeated for the subgroups gender and achievement level 
(defined as less than the median at pre-test and greater than or equal to the median at pre-test for the 
P1 marks). 


The effect size was calculated according to Hedges (2007). The 95% confidence interval (Cl) was 
calculated as the estimate +/- 1.96 SE (standard error). Table 3 summarises the primary outcomes. For 
all pupils combined, there was a moderate effect size of +0.22 which equates to around 3 months’ 
progress (Table 4). This was lower than the minimum detectable effect size (MDES) calculated in the 
power analysis (see Sample size section on p.11), suggesting that the original estimates in the 
calculation were too conservative. 


Moderate effect sizes were shown for those with lower and higher prior science attainment, with slightly 
more impact evident for those with lower attainment. There was also a moderate effect size for girls 
(+0.32).There was a lower effect size for boys (+0.12), and it is not possible to rule out chance as an 
explanation for the effect observed. . The difference between the effect on the girls and boys was 
sufficiently large to provide evidence that TDTS was more effective for girls than boys (p=.023). The 
effect size for FSM pupils was +0.38. However, it must be noted that the FSM pupils were unevenly 
distributed amongst the schools, although the numbers in the intervention and control groups were 
similar (138 control and 147 intervention). Four schools had no FSM participants and almost two-thirds 
of the 285 FSM pupils across the sample were accounted for by 11 schools. 


There were some changes to the analysis outlined in the original protocol, which stated that as well as 
the main analyses including all pupils, subgroup analyses would be carried out for boys and girls, and 
for high, average and low achievers (based on the pre-test). The analysis by achievement was divided 
into two rather than three categories (at or above the median versus below the median) to give more 
robust sample sizes. 


Analysis by implementation fidelity was not conducted because the proxy measure (attendance at 
training sessions) was high for the large majority of the sample. An additional analysis, of those ever 
eligible for FSM (to reflect the EEF’s priority focus on this demographic) could not be included because 
of low sample sizes. Ten intervention schools and 10 control schools had fewer than five FSM 
participants. The total number of FSM pupils in the analysis was 285, with eleven schools accounting 
for almost two-thirds of them. 
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Table 3: Primary outcomes 
Raw Means Effect size 
Intervention group Control group 


n Mean n Mean 
(missing) (95%Cl) 


Outcome Sy D) 


(missing) (95%Cl) 


22.25 1264 
(21.72, 6. 6.92 (655, 
22.77) 609) 


763 


All pupils (108) 


21.77 
(21.00, 6.90 6.94 (328, 
22.53) 328) 


6.90 (325, 
276) 


19.52 
(18.48, 6.38 
20.56) 


FSM 178 
pupils (31) 


Lower 
than 
median at 
pre-test 
Above or 
equal to 
median at 
pre-test 


Pupil attitudes to science 


The attitude survey was completed by Year 5 pupils around the same time as the science test, and 
findings for the control and intervention groups were compared. The survey took the form of a series of 
statements for which pupils were asked to indicate their level of agreement on a five-point scale from 
‘agree a lot’ to ‘disagree a lot’. Pupil-level results can be found in Appendix 1. Because these do not 
account for clustering within schools or the effect of pairing, Tables 4—6 (below), only provide the most 
“favourable” response (usually agree a lot, but disagree a lot where the items were reversed, as 
indicated) for intervention and control pupils separately. This is to give an indication of the strength of 
agreement or disagreement for each statement. The commentary focuses on the school level analysis 
which, although more limited, is more reliable because it allows for pairing and clustering. This was 
conducted by taking 19 pairs of schools as matched at randomisation (excluding the pair with the school 
that withdrew and one school that did not return the attitude survey). For each statement, mean scores 
were calculated for each school. The means were then compared within each pair. The tables report 
how many cases in which the intervention score was higher, lower or the same as its matched control. 


The first group of attitude statements relate to school science lessons (Table 4). For most of these items, 
the intervention schools were more favourable than the controls. This was particularly true for the 
lessons being interesting, something they looked forward to and would like to do more of. The direction 
of the response suggested intervention schools were having more discussions and doing less writing in 
science, in accordance with the TDTS model (although there was no difference between the pairs of 
schools for the likelihood of copying off the board). They were also much more likely to say science 
lessons made them think, and that they enjoyed the discussions. However, there was very little 
difference in self-reported level of understanding or enjoyment of solving problems in science. 
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Table 4: Learning science at school 
Comparison of mean item score (school- 
EAVZ=1)) 


Intervention Intervention Control 
more = Control more 
favourable favourable 


: , . TDTS 

Science lessons are interesting 

control 
Science lessons are boring* TOTS 

control 
Solving science problems is TDTS 
enjoyable Sontial 
| enjoy discussions in science TDTS 
lessons 

control 
We often have discussions in TDTS 
science lessons 

control 
We spend a lot of time in science TDTS 

1 * 

lessons copying from the board aaniel 
We do a lot of writing in science TDTS 
lessons* 

control 
Science lessons make me think 1 

control 
| understand everything in my TDTS 
science lessons 

control 
| look forward to my science TDTS 
lessons 

control 
| would like to do more science at TDTS 
school 

control 


* most favourable rating = “disagree a lot” 


= 
oa 

+ 

E 


= = 
(jo) NO 


as 
wo 


= 
wo 


ae 
= 
ye) 


nN 
fo) 

—_ 

A 


= 
wo 
= 
oa 
= 
oO 


= 
wo 


Turning specifically to the practical element of science lessons (Table 5), in all cases more intervention 
than control schools gave the most favourable responses within the pairs. In line with the TDTS 
approach, there was confirmation that practical work was done more often than in most of the control 
schools. Pupils doing TDTS were more likely to look forward to practicals and think they were fun. 
However, the “agree a lot” figures for these two statements show that practical work is a keenly 
anticipated and fun aspect of science in both sets of schools. 
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Table 5: Practical work in school science 


We do practical work in most science TDTS 
lessons control 
TDTS 
control 


| look forward to doing science 
practicals 


Doing practical work in science 


lessons is fun 


Practical work in science is boring* 


We already know what will happen 


when we do science practical work* 


| can decide what to do for myself in 


science practical work 


* most favourable rating = “disagree a lot” 


TDTS 
control 
TDTS 
control 
TDTS 
control 
TDTS 
control 
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Comparison of mean item score (school- 
EAVZ=3)) 


Control 
more 
favourable 


Intervention 
more 
icc WVolUl e= 10) (=) 


Intervention 
it Oo) ny i Ke) | 


15 


Table 6 compares attitudes of the two groups towards science more generally. The figures for pupil 
agreement show that more than half the pupils agreed ‘a lot’ that it was important to learn science, but 
within 16 of the 19 pairs of schools this was expressed more strongly in the intervention school. The 
same balance was seen for ‘science is fun’ in the context of a generally favourable response. The 
response to ‘I like thinking about scientific ideas’ was more positive in 14 intervention schools compared 
with just 4 controls. There was no real difference for science being difficult to understand, matching the 
response to ‘I understand everything in my science lessons’ in Table 4. Around two-thirds of pupils 
disagreed ‘a lot’ with the gender stereotype that science is for boys, and this was slightly more evident 


in intervention schools. 
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Table 6: Attitudes towards science 


Comparison of mean item score (school- 
EAVZ=)) 


Intervention 4 Control 
Tacs) ave) al iceyal 
more more 
it Oo) a) 10) | 
lee \WAelUle-le) (=) favourable 


TDTS 
| like thinking about scientific ideas 
control 


TDTS 
Science is fun 
control 


TDTS 
It is important that we learn science 
control 


| find science difficult to TDTS 


understand* control 


TDTS 
| am just not good at science 
control 


TDTS 
| think science is more for boys* 
control 


* most favourable rating = “disagree a lot” 
Cost 


The cost of the approach as delivered in the trial is estimated at £26 per pupil. This estimate is based 
on two classes of 25 pupils and a group of 21 schools receiving training together. Schools were charged 
£1,000 each to take part, which included five full INSET days, resources and materials. The estimate 
also includes the cost of three pilot days, which were used to refine the approach prior to the start of the 
intervention but which were not passed onto schools (estimated at £6,720 in total). 


The financial model for the evaluation was quite complex. Intervention schools were asked to provide 
funding of £1,000. Half of this was requested at the start of the project (although there was some 
discretion for schools that were unable to afford this), and half was deducted from their final payment of 
cover time. In return they received: 


e five days of CPD (plus 3 pilot days), including supply cover; 

e supply cover funding for up to two days of in-school time per teacher to plan how the project 
strategies would be best embedded in their school; 

e an equipment and resources package for each school, worth £500; and 

e acelebration and dissemination conference at the end of the year. 


Control schools were expected to make a contribution of £500 (less than intervention schools since they 
were being offered three fewer training days), which will be deducted from their final cover payment at 
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the end of the year when they receive TDTS. The package they received was the same as that outlined 
for the intervention schools, without the three pilot sessions. 
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Process evaluation 


Fidelity 


Implementation fidelity was not assessed directly through checklists or lesson observations. Instead, 
comparison was made between the survey responses of teachers from the control and intervention 
schools about the approaches used in science lessons (see below). Attendance records for the five 
training sessions that constituted the main phase of the intervention were also examined (see Table 7). 
The three pilot sessions, which took place the previous year, have not been included because in most 
schools there was significant teacher change between 2012-13 and 2013-14. In general, attendance 
was very high with only four out of the 21 schools failing to attend all five sessions (three schools 
attended four, one school attended three). 


Table 7: Teacher attendance at TDTS training sessions 2013-2014 


PN accYaletclalersy TUT an) efs) ae) meoxer alele) [oy 
(CAR) r1)) 


All sessions attended, same two teachers 


All sessions attended, same two teachers, 
occasional absence of one teacher 


All sessions attended, at least one consistent 
attendee but not always both Year 5s 


All sessions attended, not both teachers to 
every one (illness) 


Small school so only one attendee (one 
session missed) 


Some sessions missed, inconsistency in staff 
attending (high teacher turnover) 


Teacher survey 


Teacher surveys were sent to all schools in the study. In the intervention sample, all those who had 
attended training sessions were asked to return them. In control schools, all Year 5 teachers were asked 
to complete them. Thirty-three teacher surveys were received from 20 intervention schools, and 21 
surveys from 15 control schools. Although these represent very good school-level response rates (95% 
intervention and 75% control), the low number of teachers mean that the findings have to be treated as 
indicative only. 


Comparing responses from TDTS and control teachers shows that the former reported that they were 
using several approaches more often. These included practical work and pair or small group practical 
work; whole class discussion and, to a lesser extent, pair or small group discussion; and asking pupils 
to solve scientific problems. TDTS teachers were also more likely to say that they gave pupils time to 
think “very often’. These responses fitted with the pattern shown in the pupils’ responses, where those 
in intervention schools were more likely than those in controls to agree that they had practicals in most 
science lessons, often had discussions, and that the science lessons made them think. There was little 
or no difference for making links with literacy and numeracy, using teacher demonstration and teaching 
scientific facts. This suggests that TDTS prompted teachers to use more practical activities, promote 
more discussion among pupils, and give more time for reflection. 
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One of the aims of the project was to encourage teachers to make more links between science, literacy 
and numeracy. However, teachers who followed the approach did not report making more links than 
those in the control group. 


TDTS and control teachers expressed broadly similar confidence levels on various activities related to 
science teaching, with the exception of assessing pupils’ work where the TDTS teachers were more 
confident overall. There was also no apparent difference between the groups on their view of their own 
science knowledge. 


However, TDTS teachers were much more likely than control teachers to agree with a range of positive 
statements about their pupils’ ability, confidence and motivation in science over the year. The gap was 
particularly marked in relation to pupils enjoying their lessons and being engaged with science. 


When TDTS teachers were asked about the intervention specifically, around three-quarters of them 
agreed strongly that they had used the strategies they had learnt; that they had found it inspiring; and 
that it had benefited their pupils. In terms of abilities, they felt that the project had particularly helped 
them to develop their understanding of science practical work, their questioning skills and their use of 
discussion slots in science. They were least likely to agree that it had developed pupils’ recording skills 
in science, or their use of questioning skills and discussion slots in other subjects. 


Teachers spoke about being reinvigorated and inspired by the course. The strategies that they had 
found especially effective were ‘Practical Prompts for Thinking’, ‘Bright Ideas Time’ and (within that) the 
‘Big Question’. The starters were valued as a way to get children into a scientific frame of mind and 
allowing pupils of different ability levels to take part with all contributions being valued. In many cases, 
children had been given more freedom to explore and investigate, and had sometimes surprised their 
teachers: ‘Some particularly challenging questions have been asked, leading to discussions about 
science | never thought would happen in primary school! Several teachers appreciated the ideas for 
practical work and an ‘emphasis on practical science and talking as opposed to extensive work in books.’ 


The questionnaire did not specifically invite ideas for improvements, although there was a catch-all 
question at the end. Just two teachers took this opportunity to voice anything negative and both 
comments related to generic issues with science. One said that resourcing science lessons was always 
a chore, and the other bemoaned the lack of time to fit in all the topics suggested because science was 
confined to one afternoon slot a week. 


The impact of the course was summed up neatly by this teacher: ‘! have always tried to make my science 
lessons as practical as possible—but the course had given me more confidence with this approach, and 
made me realise it is not just the doing that is important, but the thinking and giving the time to talk too.’ 


Intervention teachers were asked if the TDTS approach had been particularly beneficial to certain groups 
of pupils, and control teachers were asked the same question about their usual approach. TDTS 
participants were more inclined to mention those with special educational needs (49% vs 24%). The 
intervention appeared to be judged effective across the ability range, with 55% (vs 14% control) saying 
it was beneficial for low ability pupils, 46% (vs 33% control) for middle ability, and 52% (vs 24% control) 
for high ability. 
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Conclusion 


Key conclusions 


1. Thinking, Doing, Talking Science appeared to have a positive impact on the attainment of 
pupils in science. Overall, Year 5 pupils in schools using the approach made approximately 
three additional months’ progress. 

2. There are some indications that the approach had a particularly positive effect on pupils 
eligible for free school meals, but further research is needed to explore this. 


3. The programme had a particularly positive effect on girls and on pupils with low prior 
attainment. 


4. The approach had a positive impact on pupils’ attitudes to science, science lessons, and 
practical work in particular. 

5. National test data will be used to assess the English and mathematics outcomes of 
participating pupils and to measure the long-term impact of the approach. In addition, further 
research could be conducted to investigate whether this result can be replicated in a larger 
number of schools. 


Limitations 


The trial was run in a single local authority area (Oxfordshire), so the sample of schools was drawn from 
a restricted geographical area. Moreover, the low rates of pupils eligible for free school meals in several 
participating schools means that the FSM data is based on small sample sizes. This limits the 
generalisability of the results. 


The primary outcome measure, although compiled from questions in a standardised series of 
assessment units, was not in itself a nationally standardised test. The post-test was administered 
immediately at the end of the intervention, so there is no measure of the longevity of the impact. 
Moreover, the post-test was supervised by staff who knew the treatment allocation of the schools and it 
is possible this may have introduced a source of bias. 


The process evaluation was limited to a teacher survey. It would be useful to understand more about 
which elements of the programme teachers used, and how they introduced them into their classrooms. 


Interpretation 


The results of this cluster RCT provide promising evidence that the TDTS intervention has a positive 
impact on pupils’ science attainment. The effect size was higher for girls to a degree that was unlikely 
to happen by chance, suggesting that TDTS had a greater effect on girls than boys. This impact on 
attainment was accompanied by more favourable post-intervention pupil attitudes towards science in 
TDTS schools compared with the controls, particularly in those areas in line with the project aims. This 
included science lessons being interesting, making pupils think and science being fun. The differences 
can be interpreted with reasonable confidence as being a consequence of the TDTS intervention. 


The TDTS teachers were more likely than teachers in control schools to report using the type of teaching 
approaches that were part of the intervention, although there was little conclusive evidence of increases 
in teacher confidence. Intervention teachers were, however, much more positive than their counterparts 
in the control group about their pupils’ engagement, confidence and ability in science. These findings 
were supported by the pupils’ response to the survey mentioned above. However, caution should be 
exercised when looking at these findings, both because of low sample sizes and because TDTS may 
be receiving a ‘halo effect’ from teachers excited at being involved in the intervention. 


Although the feedback received from the TDTS teachers via the survey was overwhelmingly positive, 
there remains a doubt about the extent to which the strategies have been adopted across the wider 
curriculum. The question of whether there has been any impact on pupils’ performance in literacy or 
numeracy will be explored by the analysis of the KS2 SATs results later in 2015. 


Education Endowment Foundation 26 


Thinking, Doing, Talking Science 


Future research and publications 


These uniformly positive findings suggest that TDTS is a very promising intervention. It would be an 
ideal candidate for a larger, effectiveness trial to ascertain whether such a sizeable impact can be 
maintained when the intervention is scaled up and possibly delivered using a different model. 
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Appendix 1: Pupil attitude ratings 


Learning science at school (cf Table 4) 


Agree Agree Not Disagree | Disagree 
a lot a bit sure a bit a lot 

Science lessons are interesting TDTS 49 33 8 7 3 

control 37 31 17 9 6 
Science lessons are boring TDTS 7 4 10 20 54 

control 14 16 11 19 40 
Solving science problems is TDTS 43 26 15 7 
enjoyable 

control | 36 24 18 12 
| enjoy discussions in science TDTS 32 33 19 9 8 
lessons 

control 25 28 22 13 12 
We often have discussions in TDTS 54 34 4 5 2 
science lessons 

control 41 31 18 7 3 
We spenda lot of time in science TDTS 10 17 18 29 33 
lessons copying from the board 

control 15 18 22 21 24 
We do a lot of writing in science TDTS 29 33 18 19 8 
lessons 

control 29 31 21 15 4 
Science lessons make me think TDTS 40 42 13 3 1 

control 29 46 18 5 2 
I understand everything in my TDTS 13 29 18 27 13 
science lessons 

control 13 28 23 26 11 
| look forward to my science TDTS 38 34 16 8 5 
lessons 

control 28 30 20 13 9 
| would like to do more science at TDTS 36 2A 47 13 10 
school 

control 30 20 22 12 16 
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Practical work in school science (cf Table 5) 


We do practical work in most science TDTS 26 
aoa control 17 
| look forward to doing science TDTS 53 
practicals control 46 
Doing practical work in science TDTS 61 
lessons is fun ane 49 
9 

11 

20 

16 


Agree Agree Not Disagree Disagree 


FW le) a bit sure a bit alot 


Practical work in science is boring TDTS 

control 
We already know what will happen TDTS 
when we do science practical work 

control 
| can decide what to do for myself in TDTS 
science practical work 

control 


35 
22 
22 
23 
22 
26 
15 
18 
25 
26 
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Attitudes towards science (cf Table 6) 


Agree Agree Not Disagree Disagree 
a lot a bit sure a bit a lot 
I like thinking about scientific ideas TDTS A 27 17 
control 34 23 19 
TDTS 49 30 10 
control 36 32 14 
It is important that we learn science TDTS 70 15 10 
control 57 19 15 
| find science difficult to understand | TDTS 7 34 17 24 17 
control 11 27 21 20 21 
| am just not good at science TDTS 11 15 
control 14 14 
| think science is more for boys TDTS 7 7 
control 7 5 
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Appendix 2: Content of training sessions 


Outline of Days: September 2013 — June 2014 


During this year each teacher can claim a total of 2 days each to work on this project. 


Content 


Day 1 


Theme: Materials 


Fri 13 Sept 2013 


e Anoverview of progress to date 

e  =6The Bright Ideas Time: Odd One Out at two levels - newcomers and old 
hands 

e = Challenge in science and higher order thinking contd. 

e More ideas for practical Scientific Enquiry: What’s the Best? 

e Discovery Dog/Spellbound Science 

e Drama in science 


e Apractical prompt for pondering 


Day 2 


Theme: Electricity 


Fri8 Nov 2013 


e =The Bright Ideas Time: PMI at two levels - newcomers and old hands 
e More problem solving as a means of challenge — the precious jewel 
e = Creative communication 

e —_ Literacy links 


e A practical prompt for pondering 


Day 3 


Theme: Forces and 


e  =The Bright Ideas Time: the Big Question at two levels - newcomers and 
old hands 


e More practical scientific enquiry for challenge: flippers 


motion 
e Puppets in science 
Tues 14 Jan 
e Mathematics links 
e Apractical prompt for pondering 
Day 4 e = The Bright Ideas Time: Concept Cartoons, 


Theme: Light and 


Sound 


Weds 12 March 


e Concept Cartoons as a prompt for an investigation 
e More drama in science 
e = ICT links 


e A practical prompt for pondering 


Day 5 


Theme: Evolution 


Thurs 5 June 


e = The Bright Ideas Time: Thinkers’ Keys 


e Content reviewed according to need 


Celebration and dissemination event: Friday 20 June 
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Appendix 3: School memorandum of understanding 


The Thinking, Doing and Talking Science Project 

Funded by the Education Endowment Foundation 

Delivered by Science Oxford and Oxford Brookes University and evaluated by the Institute for Effective 
Education at the University of York 


School Partnership Agreement: For all ‘intervention schools’ 
Please read & sign this agreement & return to Science Oxford at the address below by 30 April 2013. 


The project aims to work over 3 years to conduct a trial of the impact of a programme that aims to help 
primary schools to improve attainment in science by focussing on creative, practical and challenging 
science lessons that encourage pupils to use higher order thinking skills. 


This agreement is between Science Oxford and __ Primary School. 


Under this partnership agreement Science Oxford agrees to: 

e Deliver a ‘launch day’ in your school during 2013 

e Deliver 3 training days in partnership with Oxford Brookes University from February to July 2013 

e Deliver 5 training days in partnership with Oxford Brookes University during the 2013-14 school 
year 

e Deliver 2 project celebration/dissemination events in partnership with Oxford Brookes University 

e Provide supply cover funding of £200 per day for up to two teachers for each training day 

e Provide supply cover funding of £200 per day for up to two teachers to have 3 days of in-school 
planning from February 2013 to July 2014 

e Provide supply cover funding of £200 per day for up to two teachers to attend 2 project 
celebration/dissemination events 

e Supply the school with an equipment package worth £500 that complements the teacher training 


Under this partnership agreement __ Primary School agrees to: 

e Payacontribution of £1000 (£500 in April 2013 & £500 in September 2013) — or to contact 
Bridget Holligan to make an alternative arrangement if this is not possible. 

e Send two teachers (unless agreed otherwise) to each of the 8 training days arranged from 
February 2013 to June 2014 

e Use the approaches developed during the training days with 2013-14 Year 5 pupils and provide 
evaluation feedback 

e Attend the celebration/dissemination events 

e Administer the post-intervention science test (distributed and collected by the University of York) 
to all 2013-14 Year 5 pupils before the end of the school year in July 2014 


Hk Wika | 


Signature: 


€ > 


, vey 
) f tto Ux Qawn 


Print Name: 


Bridget Holligan 
Science Oxford 


Date: 25/03/2013 


Helen Wilson 
Oxford Brookes University 


Date: 25/03/2013 


___ Primary School 


Date: 
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Appendix 4: Parental consent letter 


January 2013 
Dear Parent/Guardian, 


We would like to ask permission for your child to take part in an educational 
research study. 

This study is being done to assess the effectiveness of a training course for 
teachers in helping pupils do better at science. 


Your child’s school has agreed to participate in the study. In January 2013, the 
teacher will administer short science questionnaires to the pupils in your child’s 
class. They will be tested again in June 2014. 


Your child’s answers will be confidential. They will be marked by the research 
team at York and will not count towards any school assessment. Pupils’ names 
will be replaced with code numbers. No individual pupil’s data will appear in any 
report. 


If you do not want your child to take part, please complete and sign the attached 
opt-out form by INSERT DATE. A pupil's right to withdraw will be respected. 


If you have concerns or questions about your child’s participation in this study, 
please contact Pam Hanley (e-mail: pam.hanley@york.ac.uk Tel: 01904 
328165). 


With thanks and best wishes 


Pam Hanley 
Institute for Effective Education 
University of York 
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THINKING, DOING AND TALKING SCIENCE EVALUATION 


Parent/Guardian opt-out form 

If you do not permit your child’s test scores to be used in the study, please 
complete this form and return it to your child’s teacher by INSERT DATE. 

| do not wish my child’s test scores to be used in the research project. 

PUBIES VAIN? oo cecxece cet chd Sieteeentcetesbetetecd tah nee anit katt esha ern ae ae 
(Please print clearly) 

Class T6aGnel eic62)522550 285 oe eA Ue Meee 
Parent’s/Guardian’s NAMe: .................:::ccccccccececessseseeeeecceccenssseeeeseesessanasess 


(Please print clearly) 


Parent’s/Guardian’s Signature: ..................ccccccccecccececeeeeeeeeeeeeeeeeeeeeeeeneeenaees 


Education Endowment Foundation 35 


Thinking, Doing, Talking Science 
Appendix 5: Pupil pre-test 


THE UNIVERSITY Of York 


Science knowledge 
questionnaire 


Year 4 


First name 


Last name 


School 


e The test is 40 minutes long. 

e You will need: pen, pencil, rubber, ruler, protractor and 
calculator. 

e The test starts with easier questions. 

e Try to answer all of the questions. 

e Write all your answers on the test paper - do not use any 
rough paper. 

e Check your work carefully. 

e Ask your teacher if you are not sure what to do. 


For marker’s use only TotebKiens a 
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Ql Draw a line to put each animal in the place where you could find it. One has 
been done for you. 


sea and shore farm fields 


Total out of 1 | 
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Q2 
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Look at these two objects. 


peersbare tie 


Write one way in which these two objects are different. 


Write one way in which they are the same. 
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Q3 a Put numbers on the pictures in the correct order to show how the ice ball 
changed. The first one has been done for you. 


oy 


mark 
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b What happened to the size of the ice as it melted? 

1 mark 
Cc Which word describes the water left when the ice has all melted? Tick ONE 
box. 
me] ry] 
we Tm 
Sharp || 1 mark 


Total out of 3 
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Q4 ~— Samand Anna find out how the sound changes as they move away froma 
sound maker. 


a Use these words to complete their table. 


( very quiet ) 


distance 
from sound maker 
(metres) 


2 
marks 
b Look at the table. What was the sound like at 1 metre from the sound 
maker? 
1 mark 
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Cc What would Sam and Anna use to measure how far they move from the 
sound maker? Tick ONE box. 


hand lens 


balance scale 


thermometer “ee [| 


—— 


— 


mark 


Total out of 4 
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Q5 Helen wants to make the bulb light. 


[| 


Total out of 1 [ 
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What three things do ALL animals do? Tick THREE boxes. 


move | | swim || 
reproduce | | run | | 
smile [| grow [| 
read || cry | | 


Total out of 3 | | 


Education Endowment Foundation 44 


Thinking, Doing, Talking Science 


Q7 Dylan and Gemma want to know if the root always grows downwards from a 
seed. 

They place some sunflower seeds in different positions on a paper towel. They 
water them. 

pointing downwards pointing sideways 


pointing upwards 


uN) 


vt My 
Wy) 


This is how the seeds grew. 


meet 


Complete this table to show their results. 


a 
starting position pointing pointing pointing 
of seeds upwards 
direction roots 
grew 
2 
marks 
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b Dylan said they could use just one seed pointing each way. Gemma said it was 
better to use three seeds in each position. 


Why is Gemma's suggestion better? 


1 mark 


c What did they find out about the starting position of the seeds and the 
direction in which the roots grew? 


1 mark 


Total out of 4 || 
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Q8 Peter sorted these objects into sets according to the material from which 
they are made. He called this set ‘wood’. 


table gate chair 


wood 


a Write the name of the material for this set. 


envelope 


1 mark 


b Write the name of the material for this set. 


ring 


nO 


1 mark 
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Q9 = Jack shone a torch onto some objects to see what happened to the light. 


Coin Bubbles 


Complete these sentences. Use each object only once. 


a The light went through the on...cccccsssssuescesneesseesseeneeesness 


1 mark 
b The light Made the ........ cece LOOK Shiny. 

1 mark 
Cc The light Made the ....... eee Make a dark shadow. 

1 mark 


Total out of 3 
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Q10 Colin pushed his car twice. It went much further the second time. 


Explain why the car went further on the second push. 


Tick ONE box. 


Colin pushed harder the second time. 
The car was heavier the first time. 
Cars sometimes do that. 


The wheels on the car are not straight. 


Total out of 1 | | 
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Ql1 


Which organ of the body is most likely to be damaged by too much alcohol over a 
long time? Tick ONE box. 


eye lungs 


tongue i liver _ 
<a 
Total out of 1 | 
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Qi2___—‘ The fern is a plant that grows well in woods. 


Name TWO conditions in woods which would suit ferns. 


Total out of 2 || 
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Q13. Emmaand Sam left these materials on the window sill for one week. How did 
they change? 


Draw a line from the material to the correct word. Use each word only once. 
The first one has been done for you. 


no change 


melts 


dries up 


goes soft 


Total out of 1 
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Q14 
Olivia wanted to find a good material for blocking out sounds. 
She held tubes over her ears and filled them with different materials. 
tissue cotton wool newspaper straw 
a Write down the question that she is investigating. 


1 mark 


b Michael made the sounds to see whether Olivia could hear them. Why did he 
tell Olivia to close her eyes as he made the sounds? 


1 mark 


Total out of 2 | | 
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Q15 Jane gives the same sized push to two toy cars on the table. The second car 
is much heavier than the first car. 


How will the cars move? Tick ONE box. 


The two cars move the same distance. 
The first car moves more slowly than the second car. 
The two cars move at the same speed. 
The second car does not move as far as the first car. 


You can't tell how the cars will move. 


Total out of 1 || 
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Q16 “A healthy diet contains a balance of carbohydrates, fats, proteins, vitamins, 
minerals, fibre and water." 


Explain what ‘balance’ means in this sentence. 


Total out of 1 
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Q17_ Anne and George go pond-dipping. They look for animals at the surface of 
the pond and deeper down in the pond. Here is a table of their findings. 


Animal Found at the Found deep in 
pond surface the pond 


water beetle 


leech 


pond skater 


great diving beetle 


gnat larva 


water flea 


phantom midge larva 


a Which animal did they find most of? 


b Write the name of one of the animals found only at the pond's ee iat 
1 mark 
air tube 
c Look at the drawing of the gnat larva. Why is this animal not found in the 
deep water? 
1 mark 


Education Endowment Foundation 56 


Thinking, Doing, Talking Science 


Q18 Chris and Sarah have four materials to make a temporary shelter from the 
rain. They investigate which would be the best to use to keep them dry. 


They poured the same amount of water on to each material. 


a Why did they pour the SAME AMOUNT of water on to each material? 


b What should they look for to find out whether each material is waterproof? 


1 mark 


Total out of 2 [ 
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Q19 Emma made a shadow in the shape of a rabbit on the screen. 


a Explain how a shadow is formed. 


1 mark 


b Explain how Emma uses light and shadow to make the rabbit's eye. 


1 mark 


Total out of 2 || 
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Q20_ The children investigate the time it takes different sized spinners to fall 3 
metres. 


spinner 10 
fom’) 30 
20 

la 

6 


Time. te fall 3m {s) 
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a What would be the wing area of a spinner which took 3 second to fall 3 
metres? 
cma 
1 mark 
b Describe how the wing area of the spinners affects the time taken to fall. 
2 marks 
Cc What is it that causes the area of the wing to affect the time taken for the 


spinner to fall? 


1 mark 
d Name the force which pulls down on the spinner. 
1 mark 
Total out of 5 
End of test 


Education Endowment Foundation 60 


Thinking, Doing, Talking Science 


Appendix 6: Pupil post-test 


THE UNIVERSITY Of York 


Science knowledge 


questionnaire 
Year 9 


First name 


Last name 


School 


The test is 40 minutes long 

You will need: pen, pencil, rubber, ruler, protractor and 
calculator 

The test starts with easier questions. 

Try to answer all of the questions. 

Write all your answers on the test paper - do not use any 
rough paper 

Check your work carefully 

Ask your teacher if you are not sure what to do 


For marker’s use only Total Marks Le 
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reproduce || run 
smile RR grow 
read | | cry 
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Total out of 3 | | 
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Q2 Peter sorted these objects into sets according to the material 
from which they are made. He called this set ‘wood’. 


table gate chair 


wood 


a Write the name of the material for this set. 


envelope tissue 


a 


b Write the name of the material for this set. 


1 mark 


ring coin 


1 mark 
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Q3 Dylan and Gemma want to know if the root always grows 
downwards from a seed. 


They place some sunflower seeds in different positions ona 
paper towel. They water them. 


pointing upwards pointing downwards pointing sideways 


This is how the seeds grew. 


mpp attacy 


a Complete this table to show their results. 


starting position pointing pointing pointing 
of seeds upwards 


direction roots 
grew 


2 marks 
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b Dylan said they could use just one seed pointing each way. 
Gemma said it was better to use three seeds in each 
position. 


Why is Gemma's suggestion better? 


1 mark 


c What did they find out about the starting position of the 
seeds and the direction in which the roots grew? 


1 mark 


Total out of 4 
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Olivia wanted to find a good material for blocking out 
sounds. 


She held tubes over her ears and filled them with different 
materials. 


tissue cotton wool newspaper straw 


Write down the question that she is investigating. 


1 mark 


Michael made the sounds to see whether Olivia could hear 
them. Why did he tell Olivia to close her eyes as he made 
the sounds? 


1 mark 
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Q5 Colin pushed his car twice. It went much further the 
second time. 


Explain why the car went further on the second push. 
Tick ONE box. 


Colin pushed harder the second time. 


| 
The car was heavier the first time. | | 
| | 
|| 


Cars sometimes do that. 


The wheels on the car are not straight. 


Total out of 1 |_| 
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Q6 Dan saw this warning printed ona cigarette packet. 


TOBACCO SERIOUSLY DAMAGES HEALTH 


ie 


4 
” 
4 
w 
g 
” 


Crown Cigarettes ~ 


Which body organ is most likely to be damaged by contact 
with inhaled tobacco smoke? 


Total out of 1 || 
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Q7 One year a farmer killed all the foxes on his land because 
they were killing his chickens. Later in the year he found 
that his lettuces were being eaten by rabbits. 


Why were there so many more rabbits on his land later in 
the year? 


Total out of 1 || 
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Q8 Sam and Hannah put 150g of each crushed rock into a 
separate jar. They covered each material with water. 


Next they remove the materials from the water and 
measure the mass of each again. 


a What property of the materials were they investigating? 
Tick ONE box. 


absorbency | | hardness || 


solubility rf transparency || 


b = What would they need to keep the same to make sure their 
test was fair? Tick ONE box. 


Time of day the investigation is done 
The shape of the containers 
How long the rocks are kept in the water 


The colour of the containers 


1 mark 


1 mark 
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Results 


material mass when mass when 
dry (grams) et (grams) 


sandstone 


marble 


C What was the mass of the brick when wet? 


essssssssesnsee GPOMS 


1 mark 


d = Which building material would you predict is least permeable 
to water? Tick ONE box. 


brick |_| slate | | 
sandstone |_| marble [| 


1 mark 


Total out of 4 | | 
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Q9 Which of these drawings correctly shows a boy and the 
shadow he makes standing in the playground ona sunny day? 


Tick ONE box. 


Total out of 1 | | 
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Q10 Claire and Jane test the brightness of the light with one 
bulb and then two bulbs in their circuits. 


fair? Tick TWO boxes. 


= 
ES 
[ems 
U — 


b What is the ONE and ONLY thing they should change as 
they carry out their test? 


1 mark 
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Ql11 Yeast is a micro-organism that is useful to humans. 
Tick ONE correct way in which it is used. 


Spread on cuts 

Use in baking bread 
Use for sunburn 
Makes hair grow 


Use for plant fertiliser 
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la 


Total out of 1 || 
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Q12 Look at the photograph of the flower below. 


Label the part of the flower at A. 


1 mark 
Seeds need water to germinate. Yet in Britain many seeds 
produced by wild plants in the autumn do not germinate until 
the following spring. Why do seeds germinate and grow 
better in spring? 


1 mark 
What must happen to the pollen in order for a flower to be 
pollinated? 


1 mark 


Total out of 3 
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Q13 Look at the table below. Some of these changes are 
reversible and others are not. 


Complete the table to show the reversible changes and the 
irreversible changes. 


The first one has been done for you. 


Reversible OR 
Irreversible 
Change change 


Burning paper 


Melting an ice cube 


Boiling an egg 


CoD, des 
| ILI LI * 


Stretching elastic 


Total out of 1 | | 
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Q14 Megan is plucking a string of her guitar to make a sound. 


a How does plucking the string cause it to make a sound? 


1 mark 


b Tick TWO boxes to show how she could make a sound witha 
higher pitch. 


She could pluck the same string harder. || 
She could pluck the same string more softly. | | 
She could tighten the same string before she plucks it. | | 
She could pluck one of the thicker strings. | | 

= 


She could pluck one of the thinner strings. 
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Q15 The arrows on the photograph show two of the forces 
acting on the block when it is being pulled across the table. 


Complete the following sentences to name the forces A and 
B. 


a Force A is 


1 mark 


b Force B is 


1 mark 


Total out of 2 || 
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Q16 One of the functions of your skeleton is to protect some of 
your body's organs. 


Complete these sentences. 


QA  ——-Yoourr skull protects Your o......ccccccscssssessssssssesesssescsssseseseesescereesen 


Bh OU erea recieeemiqiemaeeeauseneecers protect your heart and 
your lungs. 


Total out of 1 || 
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Q17 Look at the two photographs of the same part of a garden 
where an onion had been planted. 
All the weeds were cleared. 


second 
photo 


The second photograph was taken after one week. 
Many small plants have grown, although none of them were 
planted. 


Where did all the small plants come from? 


Total out of 1 
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Q18 The children observe water droplets forming inside the 
window. 


a What is the name of the process by which droplets of water 
form inside the window? 


1 mark 


b Where do the droplets of water come from? 


1 mark 


Total out of 2 
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Q19 The children investigate the effect of changing the angle of 
the slope on the distance the car travels. 


They want to be sure that their test is fair. 


a = What is the ONE and ONLY thing they should change as 
they do their investigation? 


1 mark 


b Write TWO things they should keep the same. 
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Cc Results 


angle of slope distance travelled 
(degrees) (cm) 


They made a line graph of the results. 


Use the graph to predict the angle of the slope which would make 


the car travel 200cm. 
oes, degrees 


1 mark 


d Mary suggests that they should measure the distance 
travelled three times for each angle. 
Why is it a good idea to measure the distance three times? 


1 mark 
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Q20 Yasmin can see the bulb reflected in the mirror. 


Draw arrows on the picture to show how light travels to 
allow her to see the reflection of the bulb. 


End of test 
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Appendix 7: Pupil survey 


What do you 
think of 


O 


O 


O 
This booklet asks questions about you and your interest in science. 


There are no right or wrong answers. We want to know what you think. 


My name is 


lama D girl Oi boy 


My teacher is 


My school is 


wesssetzes:tOllow Science Runner........:: 
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Q1 Learning science at school 
Do you agree with these views? 


(Please tick only one box in each row.) 


Agree Agree Disagree Disagree a 
alot abit Notsure  apit lot 
Science lessons make me think. 
TCR NRE cera i iret orem HO OU O O U 
b) | look forward to my science lessons. 
oe MSM SOG. gee at oN Rr oh Bick tia OU U U U 
Science lessons are interesting. 
Sea bea aN Bd ete el ac Pana El. JE O UO U 
d) | would like to do more science at school 
paste ashsslee et teat tha in eae tednatas OU U U U 
e) Science is 
fun 
scteaasaissneieasteuteqnttiesta sh tettetepataa ucteante i ae O U U 
f) We spend a lot of time in science lessons 
copying from the board.................:::006 El LI i U ie 


g) | enjoy discussions in science lessons...... 


h) Science lessons are boring...........00000.0. 


Well done! 
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rm 
Now off you go to the next one.........02085 
Q2 Learning science at school 
Do you agree with the following statements? 
(Please tick only one box in each row.) 
Agree Agree Disagree Disagree a 
alot abit Notsure  apit lot 
| find science difficult to understand. 
yg elise os ies fe desta divi coaondinleet OU U U L 
b) | am just not good at 
science. 
c) | think science is more for boySs.............+ 
d) | understand everything in my science 
lessons 
se risa tila Sicetts 8 Pssludaebieltalesttrcer cada calla OU O U U 
e) We often have discussions in science 
lessons 
slececee testers dceotinne camncaiet one Geed O O L L 
f) We do a lot of writing in science lessons... O O O C] C] 
9) It is important that we learn science.......... O O OC O O 
h) | like thinking about scientific ideas........... O O O O O 
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Phew! Nearly finished ... 
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Q3 About practical work in school science 
Do you agree with these views? 


(Please tick only one box in each row.) 


Agree Agree Disagree Disagree 
alot abit Notsure a bit alot 
Doing practical work in science lessons is 
a) fun 
Ne bac bates celica tale ecg OU U U U 
b) We already know what will happen when 
we do science practical 
work 
oe cereale Rececgees eed deceeede tae sheet OU L L i 
| can decide what to do for myself in science 
Cc) practical WOFK..........cccceeeeees O O LO is] LO 
d) We do practical work in most science 
lessons 
A Sener an Ree Cen en U O U U U 
e) | look forward to doing science 
practicals 
seed tes th tat Resto g thacatacet bate doah, Ss a fecsiasets fay 2 a U 
f) Practical work in science is boring......... O il O 
g) Solving science problems is enjoyable...... L] C] C] 


Q4 Finally, is there anything you would like to say about your science lessons this year? 


Did you answer every one? You did? Then, you have finished ! 


Science Runner says 
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Thank you very much Y and good luck! * 
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Oxford Thinking, Doing and Talking Science evaluation 


Teacher survey 


Please describe your role in the school (you can tick more than one): 
O Year 5 teacher 
O Teacher of another KS2 year group 
1 Teacher of a KS1 year group 
O1 Science co-ordinator 
1 Other (please write in) 
How often do you teach science to Year 5 pupils? 
1 once aweek or more often 
O once a fortnight 
O oncea month 
1 less often 
O never 
How often do you do the following in science lessons? 
Very Quite Not very Rarely Never 
often often often 
a) | Pupil practical work CO CO a O O 
b) | Teacher demonstration | | CL CL C 
c) | Pupil discussion in whole class Cc Cc] Cc Cc Cc 
d) | Pair or small group practical work Cc Cc Cc Cc Cl 
e) | Pair or small group discussion Cc Cc] Cc Cc Cl 
f) | Give pupils time just to think ie) C C C i 
g) | Make links with literacy and numeracy CT CT] Cc CT Cl 
h) | Teach scientific facts C C C C ie 
i) | Ask pupils to solve scientific problems C C C C | 
How confident do you feel doing the following? 
Very Quite Notvery | Not at all 
confident | confident | confident | confident 
a) | Teaching science overall CT] Cc CT Cl 
b) | Planning science lessons CT Cc Cc Cl 
c) | Assessing pupils’ science work C C C CT 
d) | Running science practicals Cc Cc CT Cl 
e) | Teaching scientific facts C C C Cc 
f) | Explaining scientific ideas C C C Cc 
g) | Showing pupils how to record their practicals CT] CT Cc] Cl 
h) | Helping pupils discuss scientific ideas CO CO CO oO 


Education Endowment Foundation 


91 


5. 


6. 


7. 


Thinking, Doing, Talking Science 


Do you think your approach to teaching science is particularly beneficial for certain groups 


of pupils? Please tick which one(s): 


English as an Additional Language (EAL) 
Special Educational Needs (SEN) 
Disadvantaged pupils 

Boys 

Girls 


OOoOoOdo 


OOOoOdO 


Low ability 


Middle ability 


High ability 


None in particular 


Other (please write in): 


Do you find any particular groups of pupils struggle with your approach to teaching 


science? Please tick which one(s): 


C1 English as an Additional Language (EAL) OO Low ability 

CO Special Educational Needs (SEN) O Middle ability 

CO Disadvantaged pupils 0 High ability 

OO Boys CO None in particular 

O Girls 11 Other (please write in): 

Reflecting on your experience of teaching science this year, how much would you agree or 

disagree with the following statements? 
Agree Agree Disagree | Disagree 
strongly | slightly slightly strongly 
a) | My pupils have enjoyed their science lessons Cc] Cc Cc] Cc 
b) | My pupils have made good progress in science Cc Cc Cc] Cc 
c) | My pupils are confident in science C a C C 
d) | My pupils can work on their own in science Cc Cl Cc Cc 
e) | My pupils come up with their own scientific ideas (a) Cl CT CT 
f) | My pupils do a lot of writing in science C Cc C C 
g) | | have changed the way | teach science Cc Cl CT CT 
h) | | enjoy teaching science [=] Cl Cc [es 
i) | | have a good knowledge of science C CT C C 
j) My pupils have been engaged with science [=| Cl Cc CT 
Education Endowment Foundation 92 


Thinking, Doing, Talking Science 


These questions are specifically about the Thinking, Doing, Talking Science project that 
your school has been involved in. [NB Qs 8-12 asked of intervention teachers only] 


8. Are you aware of the Thinking, Doing, Talking Science project? 


C1 Yes — please answer all following questions 
CO No-please skip to Q13 


since February 2013 have you personally attended? 
O none 

one or two 

a few (3-5) 

most (6-7) 


O 
O 
O 
OO all (8) 


How many of the 8 days training involved in the Thinking, Doing, Talking Science project 


10. How much would you agree or disagree with the following statements about the Thinking, Doing, Talking 


Science project and strategies? 


Agree Agree Disagree | Disagree 

strongly slightly slightly strongly 
a) | | have used the strategies | learnt [] Ry Ey Cl 
b) | It has made me more effective teaching science Cc a C Cc 
c) | | learnt nothing new f= C fal CO 
d) | | found it inspiring [ial C C Cc 
e) | It has been useful in subjects other than science Ed i Fl fe] 
f) | It has benefited my pupils C C fe C 

11. How much has the Thinking, Doing, Talking Science project helped you develop the following? 
A lot Quite a Not very | Not atall 
lot much 

a) | My questioning skills in science Cc] Cc [es] Cc 
b) | My questioning skills in other subjects Cc Cl Cc CT 
c) | My use of discussion slots in science C CT Cc C 
d) | My use of discussion slots in other subjects C [4 [a] C 
e) | Pupils’ recording skills in science C CT Cc C 
f) | My understanding of science practical work CO fy oO | 
g) | My ownscience subject knowledge | C C | 
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12. 


13. 


14. 


Thinking, Doing, Talking Science 


Which of the strategies that the project has promoted have you found to be the most effective, and why? 


Finally, is there anything else you want to say about your experience of the teaching and learning of science 
this year? [NB: asked of all teachers] 


Name of your school: 


Many thanks for your help. Please return this questionnaire to the IEE with the pupil papers. 
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Appendix 9: Security classification of trial findings 


: : 5. Threats t 
1.Design: ee 3. Attrition: 4. Balance: Seer 
What is the What is the Adjustment valtalty: 


minimum Adjustment to 


uality of the EW7=) Me) mel ge) to rating to 
9 Y detectable P e account for issues 


design of the out from the account for 


betnns effect at the Kintion? balanted with 

Vi ions a ner f - 

chee bier hd start? hick ioe interpretation? 
Evaluation design Implementation Analysis and interpretation 


1. Design 3. Attrition 4. Balance 5. Threats to 
i. Design 9. AtrItON 4. balance 9. Inreats to 


MDES validity 


Well-balanced on 


Well-matched comparison 
(quasi-experiment) 


Matched comparison 
(quasi-experiment) 
Comparison group with | | 
poor or no matching y y 
No comparator Imbalanced on a 


The final security rating for this trial is 3@. This means that the conclusions have moderate security. 


e This evaluation was designed as a randomised controlled trial. 

e The sample size was designed to detect a MDES of less than 0.3, by design, reducing the 
security rating to 4 @. 

e At the unit of randomisation (school), there was very low attrition (1 school out of 42). 

e Balance at baseline was high. 

e The post-tests were administered by the schools by teachers who were aware of the 
treatment allocation, reducing the security of the findings by a further padlock. 

Therefore, the final security rating is 3 @. 
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Appendix 10: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found on the 
EEF website. Cost ratings are awarded as follows: 


Cost rating Description 


£ Very low: less than £80 per pupil per year. 
EE Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per 
year. 


ELEEE High: up to £1,200 per pupil per year. 
ELEEELE Very high: over £1,200 per pupil per year. 
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Appendix 11: Analysis using KS2 reading and maths data 


In December 2015, using the May 2015 Key Stage 2 reading and mathematics test data from the trial 
cohort, further analysis was conducted. The hierarchical linear modelling (HLM) analysis, a multilevel 
analysis in which students are nested within schools, was repeated with the same source of pre-test 
data (the science test), but replacing the post-test science outcome measure with (a) the fine grade 
point score for KS2 reading, and (b) the fine grade point score for KS2 Maths. Fine grade point scores 
are based on the underlying marks for the KS2 tests and indicate more precisely where the performance 
lies between the broader thresholds (ie, level 4, 5, 6 etc). This was conducted on 41 out of a possible 
42 schools. The control school that dropped out of the study could not be included in this analysis 
because it had not provided the necessary information to apply for pupil KS2 scores. 


Table A1: Numbers of participants (KS2 reading and mathematics) 


Intervention — analysed 21 699 
Intervention — not analysed 0 64 
Control - analysed 20 656 
Control - not analysed 1 94 


The two treatments were compared, and subgroup analyses were carried out for boys and girls; and 
for high and low achievers based on the pre-test scores (using ‘greater or equal to the median’ and 
‘below the median’ as the categories). We conducted an analysis for FSM-eligible pupils (defined as 
ever eligible for FSM), although — as in the main report — caution should be exercised since there were 
some schools with such low rates of FSM pupils that they had to be excluded from the analysis. In the 
main trial, ten intervention schools and 10 control schools had fewer than five FSM participants. The 
total number of FSM pupils in the original analysis was 285, with eleven schools accounting for almost 
two-thirds of them. 


The effect size was calculated using Hedges’ g. To be consistent with the main analysis, the effect size 
was Calculated as the adjusted difference in the means between the control group and treatment group 
(accounting for pairing) divided by the unadjusted total standard deviation in the outcome at endline. 


The findings show no evidence of an impact of Thinking, Doing, Talking Science (TDTS) on pupil 
performance in KS2 maths or reading tests (Tables A2 and A3) compared to business as usual. This is 
also true for the analysed sub-groups. The overall mean for reading was 28.66 for the intervention group 
and 28.84 for the control; for maths the figures were 29.03 and 29.50 respectively. Statistically, F tests 
the effect of the treatment using pairwise comparisons of the estimated marginal means within the two 
groups. Calculations of F showed that differences between TDTS and control group performance were 
likely to have happened by chance [Reading F(1, 36.8) = 0.02, p = .903 and Maths F(1, 35.5) = 0.57, p 
= 455]. 


This suggests that the impact of TDTS on science failed to transfer into English and Maths, as measured 
by KS2 performance. However, a year had elapsed since the end of the evaluation period and the 
division between intervention and control groups may have become blurred during that time. Firstly, it 
is uncertain how many pupils in the intervention schools continued to receive TDTS in Year 6 (this 
would be dictated primarily by how many of their teachers were trained either in the original study or by 
other teachers in the school). Moreover, teachers in the control schools were offered TDTS training 
from September 2014 and, although they were asked not to use it with the trial cohort in Year 6, this 
may not always have been the case. 
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Table A2: Secondary outcome - KS2 Reading Fine Grades 


Raw Means Effect size 
Taivcvav(clartcela me | cele e) Control group | 
Outcome n Mean £51 B) n Mean SD nin Effect 
(missing)  (95%Cl) (missing) (95%Cl) Harel (=1| size 
‘(Tala (95%Cl) 
(efoJ))] 
All pupils 763 28.66 4.28 750 28.84 4.18 699, -.01 
(64) (28.34, (94) (28.52, 656 (-.12, .10) 
28.98) 29.16) 
Boys 367 28.20 4.59 374 28.67 4.23 355, -.06 
(12) (27.72, (16) (28.23, 358 (-.20, .09) 
28.68) 29.11) 
Girls 359 29.12 3.88 311 29.05 4.11 342, .02 
(17) (28.71, (13) (28.58, 298 (-.14, .17) 
29.53) 29.51) 
FSM 178 27.00 4.34 161 27.71 4.04 171, 18 
pupils (7) (26.99, (8) Gro 153 (-.40, .04) 
27.66) 28.35) —* 


Lower 353 26.24 4.24 349 26.48 4.23 324, -.09 
than (29) (25.78, (44) (26.00, 305 (-.25, .06) 
median at 26.70) 26.95) 
pre-test 
Above or 410 30.75 3.02 401 30.90 2.83 375, .01 
equal to (35) (30.44, (50) (30.60, 351 (-.13, .16) 
median at 31.06) 31.20) 
pre-test 
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Table A3: Secondary outcome - KS2 Maths Fine Grades 


Raw Means Effect size 
Intervention group @xo}n tx) me] xeu] 0) 
Outcome n Mean SB) n Mean Effect 
(missing) (95%Cl) CTare)) (95%Cl) size 
(ey/1e3)) 
All pupils 763 29.03 5.10 750 29.50 4.95 699, -.06 
(64) (28.65, (94) (29.13, 656 (-.16, .05) 
29.41) 29.88) 
Boys 367 29.56 5.22 374 30.31 4.96 355, -.13 
(12) (29.01, (16) (29.79, 358 (-.28, .02) 
30.10) 30.82) 
Girls 359 28.46 4.90 311 28.54 4.77 342, -.00 
(17) (27.94, (13) (27.99, 298 (-.16, .15) 
28.98) 29.08) 
ESM 178 26.86 4.56 161 27.61 4.16 171, 47 
pupils (7) (ote, (8) (26.95, 153 (-.39, .05) 
27.55) 28.28) — 


Lower 353 26.03 4.29 349 26.78 4.38 324, -.14 
than (29) (25.57, (44) (26.28, 305 (-.30, .01) 
median at 26.50) 27.27) 
pre-test 
Above or 410 31.62 4.25 401 31.88 4.14 375, -.03 
equal to (35) (31.19, (50) (31.44, 351 (-.17, .12) 
median at 32.05) 32.31) 
pre-test 
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